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SYSTEM AND METHOD FOR AUTOMATIC NEGOTIATION OF A 
SECURITY PROTOCOL 

CROSS-REFERENCE TO RELATED APPLICATION 

[0001] Not applicable. 

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR 

DEVELOPMENT 

[0002] Not applicable. 

FIELD OF THE INVENTION 

[0003] The invention relates to the field of networked computing, and more 

particularly to the automatic negotiation of security protocols between a security- 
enabled domain and one or more external nodes. 

BACKGROUND OF THE INVENTION 

[0004] Advances in networking technology have permitted network administrators 

and others to maintain greater and more sophisticated security controls on their 
networks and other installations. Microsoft Windows™ NT, 2000 and related 
products for instance permit administrators to deploy security-enabled network 
domains using the Active Directory™ (AD) structure. The publicly known Kerberos 
network standard likewise permits nodes within a network to authenticate each other, 
using a key/authentication platform. With these operating technologies, a network 
administrator may be able, for instance, to push rules, applications, patches, drives 
and other resources from a network server to individual workstations or other clients 



Docket No. 303401.01 
MFCP. 102772 

for uniform installation, on a secure basis. All machines within the security-enabled 
domain may be able to identify and authenticate the transmission of those and other 
types of data, transparently. 

[0005] However, the ability to deliver rules, applications or other resources to and 

from a workstation becomes more difficult when that node lies outside the security- 
enabled domain. For instance, a company may have a collection of computers located 
on a local area network (LAN) but also interact with computers in a remote location 
which are not part of the Active Directory™ or other security-enabled domain. 
Communicating across the boundary of a secure domain becomes more complicated, 
in part because establishing a connection between a machine internal to the domain a 
machine outside the domain requires that an agreement be reached on a mutually 
supported security protocol. 

[0006] Systems administrators and others are therefore forced to attempt to arrange 

for the entry of an external agent or node into the security-enable domain by 
identifying a compatible protocol between the internal and external machines, before 
the session takes place. For instance, an external node may be configured to 
communicate via a transport layer security (TLS) protocol, a Kerberos-based protocol, 
a secure socket layer (SSL) or other protocol with an administrative server within the 
security-enabled domain. That machine may in turn may in that protocol its default 
protocol, indicate a protocol failure, request that the protocol be switched, or make 
other responses to the external node or agent. Manual setting or adjusting of the 
security, transport and other protocols may therefore be required, a process which 
may be time consuming and prone to error. Other problems exist. 



Docket No. 303401.01 
MFCP. 102772 

SUMMARY OF THE INVENTION 

[0007] The invention overcoming these and other problems in the art relates in one 

regard to a system and method for automatic negotiation of a security protocol, in 
which secure communications with an external agent or node may be established and 
identities authenticated, on an automated basis without a need for administrator 
intervention. According to the invention in one regard, a network manager or other 
agent or node within a security-enabled domain may initiate an attempt to establish a 
secure connection with an external agent or node. That request may contain a data 
field indicating a set of security protocols available for use by the manager. The 
external agent may receive the request and compare the protocols available to the 
. internal agent or manager to a set of protocols supported by the external agent. If a 
match between available protocols is found, communications may proceed based on 
that selected protocol. In embodiments, each of the external agent and internal agent 
may authenticate each other, via a key, certificate, or other authentication mechanism. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0008] Fig. 1 illustrates a network architecture in which an embodiment of the 

invention may operate. 

[0009] Fig. 2 illustrates a negotiation process between an internal node and an 

external node, according to an embodiment of the invention. 

[0010] Fig. 3 illustrates a comparison between protocol tables, according to an 

embodiment of the invention. 
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[0011] Fig. 4 illustrates overall protocol negotiation processing, according to an 

embodiment of the invention. 

DETAILED DESCRIPTION OF EMBODIMENTS 

[0012] Fig. 1 illustrates an architecture in which a protocol negotiation platform and 

method may operate, according to an embodiment of the invention. As illustrated, in 
the illustrated embodiment a set of clients, servers, agents or other nodes or machines 
may operate in a security-enabled domain 102. Security-enabled domain may in 
embodiments be or include, for instance, Microsoft Windows Active Directory , a 
Kerberos or other certificate-based or key-based domain, or other closed or secure 
distributed directory or other environment. Illustratively shown within security- 
enabled domain are an internal manager 104, which in embodiments may be or 
include a server or other node, as well as a set of internal agents 106 (illustrated as 
Al, A2 ... AN, N arbitrary). 

[0013] In embodiments the set of internal agents 106 may consist of or include 

additional servers, workstations or other clients, or other internal agents or nodes 
operating within the security-enabled domain 102 and communicating with internal 
manager 104. In embodiments the internal manager 104 may schedule or perform 
network administrative functions, such as transmitting or "pushing" network rules or 
other data to the set of internal agents 106, such as operating guidelines for storage 
(e.g. RAID policies, failover criteria, memory limits), bandwidth utilization or other 
rules or data. When communicating these or other types of data, the internal manager 
104 and set of internal agents 106 may take advantage of the security resources of 
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security-enabled domain to ensure the integrity of the network and the distribution of 
rules and other data. 

[0014] As illustrated, in embodiments the security-enabled domain 102 may provide 

authentication services, for instance using certificates such as certificate 108, which 
may in embodiments be or include as a certificate configured according to X.509 or 
other standards or formats. In embodiments keys or other mechanisms may likewise 
be used. As illustrated, certificate 108 may be associated with and provide 
authentication data for the internal manager 104. Any one of the set of internal agents 
106 may authenticate the rules, instructions or other data received from the internal 
manager 104 by communicating certificate 108 to a certificate authority 110 for 
verification. Certificate authority 110 may itself be located within security-enabled 
domain 102, or as illustrated be located outside the security-enabled domain 102. 

[0015] In embodiments, the certificate authority 110 may be or include a server or 

other node configured to read and decode certificate 108 or other authentication 
mechanisms, and return results to the set of internal agents 106 or other nodes. Each 
of the nodes in the set of internal agents 106 may likewise have associated with them 
a certificate, key or other authentication data compatible with the security-enabled 
domain 102. Nodes in the set of internal agents 106 may likewise communicate with 
and mutually authenticate each other, using certificate or other mechanisms. 

[0016] In the embodiment illustrated in Fig. 1, an external agent 1 14 may likewise be 

configured to communicate with internal manager 104 via communications network 
112. The external agent 114 may also be or include a server, workstation or other 
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node or resource. The external agent 114 may likewise have associated with it a 
certificate 116 identifying the external agent 114 for authentication. The 
communications network 112 through which external agent 114 may communicate 
with internal manager 104 or other internal nodes in embodiments may be, include or 
interface to any one or more of, for instance, the Internet, an intranet, a local area 
network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a 
storage area network (SAN), a frame relay connection, an Advanced Intelligent 
Network (AIN) connection, a synchronous optical network (SONET) connection, a 
digital Tl, T3, El or E3 line, Digital Data Service (DDS) connection, an ATM 
(Asynchronous Transfer Mode) connection, an FDDI (Fiber Distributed Data 
Interface), CDDI (Copper Distributed Data Interface) or other wired, wireless or 
optical connection. The external agent 114 may in embodiments be or include a 
workstation, server, wireless network-enabled device, or other node, agent or platform 
configured for networked communications. 

[0017] Unlike prior implementations of cross-domain communication, according to 

embodiments of the invention the external agent 114 may initiate' contact with the 
internal manager 104 to establish a secure connection based on a mutually compatible 
protocol with manually selecting a compatible protocol, in an automatic and 
transparent fashion. As illustrated for instance in Fig. 2, an external application 130 
executing on external agent 114 may initiate contact with internal manager 104 via 
external negotiation engine 126. External application 130 may be or include a 
systems utility, productivity or other application, such as, for instance, a data backup 
scheduler, a firewall, virus protection or other application. External application 130 
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may for example require user profiles, updates or other data to perform various tasks 
and therefore initiate such communication with internal manager 104. 

[0018] The external negotiation engine 126 may process and manage the 

communication requested by the external application 130, to establish a mutually 
compatible communications link to the internal manager 104 in security-enabled 
domain 102. As illustrated, in embodiments the external negotiation engine 126 may 
initiate and manage a negotiation module 118, illustrated as an implementation of the 
publicly known Simple and Protected GSS-API Negotiation (SPNEGO) protocol. 
Other protocols may be used. In embodiments, negotiation module 118 may be 
accessed, initiated or generated via an operating system of external agent 114, for 
instance via an application programming interface (API) or other mechanisms. 

[0019] The external negotiation engine 126 may likewise include or generate an 

external transport specifier 120 indicating a message-based or other channel which 
external agent 114 may employ to execute the protocol negotiation process. For 
instance, in embodiments the external transport specifier 120 may specify a Security 
Support Provider Interface (SSPI) protocol, as part of the Microsoft .NET 
architecture, permitting external application 130 or other software or modules to 
access for instance dynamic link libraries (dlls) or other resources supporting standard 
cryptographic or other encoding schemes. Other protocols may be used or specified 
in external transport specifier 120. The external negotiation engine 126 may 
consequently communicate a datagram indicating that or other data to an internal 
negotiation engine 128 associated with internal manager 104, as illustrated in Fig. 2. 



-7- 



Docket No. 303401.01 
MFCP. 102772 

[0020] Internal negotiation engine 128 may likewise include or interface to a. 

negotiation module 122 and internal transport specifier 124. Internal negotiation 
engine 128 may in turn communicate with an internal application 132 executing on or 
accessed by internal manager 104. Internal application 132 may, for example, be or 
include a systems administration, productivity or other application. Upon receipt of a 
request to establish communication with the internal manager 104, the internal 
negotiation engine 128 may establish a message-based or other channel with external 
agent 114 via internal transport specifier 124, for instance confirming channel 
communications using the SSPI protocol. 

[0021] With a preliminary channel established between external agent 114 and the 

internal manager 104, the external negotiation engine 126 and external negotiation 
engine 128 may initiate protocol negotiation and reduction. In embodiments, the 
external agent 1 14 may transmit an external protocol table 134 as illustrated in Fig. 3 
to the internal manager 104. The external protocol table 134 may specify which 
protocols external agent 114 may be configured to use. When received by the internal 
manager 104, the external protocol table 134 may be compared to an internal protocol 
table 136, indicating a set of security protocols available for use by internal manager 
104. Either one or external protocol table 134 and internal protocol table 136 may 
include fields indicating, for example, transport layer security (TLS), secure socket 
layer (SSL), Kerberos, secure IP (IPSec) or other available protocols or standards. 
The negotiation engine 128 associated with the internal manager 104 may identify one 
or more protocols mutually supported by external agent 114 and internal manager 
104, as illustrated in Fig. 3. 
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[0022] Negotiation engine 128 may in embodiments likewise communicate internal 

protocol table 136 to the negotiation engine 126 associated with external agent 114, 
for similar protocol comparison. Negotiation engine 126 and negotiation engine 128 
may consequently negotiate the selection of a mutually available protocol to establish 
secure communications across security-enabled domain. For instance, if only a single 
common protocol is available to both external agent and internal manager 104, the 
external agent 114 and the internal manager 104 may agree to set up a session using 
that protocol, such as TLS or another protocol. If the negotiation engine 126 and 
negotiation engine 128 agree that no common protocol may be found, the attempt to 
establish cross-domain communications may be terminated. Conversely, if the 
negotiation engine 126 and negotiation engine 128 identify multiple protocols in 
common, a protocol may be selected based on network criteria, such as transfer speed, 
bit depth of keys or other security mechanisms, or other factors. 

[0023] With a mutually compatible protocol in place, a secure session between 

external agent 1 14 and internal manager 104 may be established. In embodiments, for 
added security each of external agent 1 14 and internal manager may likewise perform 
authentication steps to verify the identity, privilege level or other security details of 
the opposite node. As illustrated in Fig. 1, this may be performed using certificates or 
other security mechanisms. External agent 114 may authenticate internal manager 
104 by communicating certificate 108 to certificate authority 1 10. Internal manager 
104 may conversely authenticate external agent 1 14 by communicating certificate 1 16 
to certificate authority 1 10. Other security mechanisms may be used. 
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[0024] The type or content of data exchanged between the external agent 114 and 

internal manager 104 may in embodiments depend on the mutual authentication 
between the two nodes. For instance, access to network administrative rules or 
parameters may be reserved for internal or external nodes only indicating a given 
level of access privilege. Other authentication rules or criteria may be used. After the 
operational security protocol has been established and any authentication processing 
is complete, the external agent 114 and internal manager 104 may exchange data, 
applications, rules or other information. When the traffic is complete, negotiation 
engine 126 and negotiation engine 128 may release or terminate the communications 
link. 

[0025] Overall network negotiation processing according to an embodiment of the 

invention is illustrated in Fig. 4. In step 402, processing may begin. In step 404, a 
request to establish a secure connection across the security-enabled network 102 may. 
be generated in either of external agent 114, internal manager 104 or other clients, 
agents or nodes. In step 406, the request to establish a secure connection may be 
transmitted to the recipient node, whether internal manager 104, external agent 1 14 or 
another client, agent or node, the request incorporating a first protocol set compatible 
with the transmitting node. In step 408, the request may be received by the recipient 
node. In step 410, the recipient node whether internal manager 104, external agent 
1 14 or another client, agent or node may compare the first protocol set with a second 
protocol set of the recipient node, to determine if a match may be found amongst 
available protocols. 
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[0026] If a match is found between the first protocol set and the second protocol set, 

processing may proceed to step 412 where a determination may be made whether 
more than one matching protocol has been found. If more than one matching protocol 
set has been found, processing may proceed to step 414 where one of the matching 
protocols may be selected for use based on protocol criteria, such as transfer speed, bit 
depth of keys or other security mechanisms, or other factors. Processing may then 
proceed to step 416, where a secure connection or session may be initiated between 
external agent 114 and the internal manager 104, based on the selected protocol. 
Likewise, if in step 412 only one matching protocol is found, processing may proceed 
to step 416 where a secure connection or session may be initiated. For instance, in 
embodiments specified ports may be opened under the TCP/IP or other 
communication or other protocols. 

[0027] In step 418, a protocol-specific exchange may be initiated between the 

external agent 114 and internal manager 104, with handshaking and other steps 
proceeding according to the matching protocol employed. In step 420, either one of 
external agent 114 and internal manager 104 or both may authenticate the 
corresponding other node by transmitting the corresponding certificate 116 (of the 
external agent 114) or certificate 108 (of the internal manager) to certificate authority 
108, as appropriate. In embodiments, the certificate 1 16 or certificate 108 or other 
security data may be or include certificate objects conforming to the X.509 standard, 
or other standards or. formats. With appropriate authentication complete, processing 
may proceed to step 422, in which a secure connection or session may be conducted 
between external agent 1 14 and internal manager 104. For instance, network or other 
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rules may be communicated between the two nodes, for systems administration or 
other purposes. 

[0028] When the secure session is complete, processing may proceed to step 424 

where the secure connection between the external agent 1 14 and internal manager 104 
may be terminated or released. In step 426, processing may terminate, repeat, return 
to a prior processing point or take other action. Likewise if no matching protocol may 
be identified in the determination of step 410, processing may proceed to step 426 to 
terminate, repeat, return to a prior processing point or take other action. 

[0029] The foregoing description of the invention is illustrative, and modifications in 

configuration and implementation will occur to persons skilled in the art. For 
instance, while the invention has generally been described in terms of a single 
external agent 114, in embodiments multiple external agents or nodes may be 
configured to automatically negotiate a matching protocol with internal manager 104 
or other clients or nodes within security-enabled domain 102. Similarly, while an 
authentication mechanism has generally been described as being supported by a single 
authentication entity 110 using X.509 or other standards, in embodiments multiple 
authentication entities or other authentication or authorization platforms may be used. 
Other hardware, software or other resources described as singular may in 
embodiments be distributed, and similarly in embodiments resources described as 
distributed may be combined. 

[0030] Moreover, while instances in which one or the other of nodes or agents 

external to the security-enabled domain 102 and nodes or agents internal to that 
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domain have been described at times as initiating the negotiation of a secure protocol, 
it will be understood that any node or agent configured according to the invention, 
external or internal to the domain, may initiate protocol processing. Likewise either 
one or both of internal and external agents may initiate authentication of the opposite 
agent or node. The scope of the invention is accordingly intended to be limited only 
by the following claims. 



